Multi-label Learning Approach Research Articles

A multi-label transformer-based deep learning approach to predict focal visual field progression

A multi-label transformer-based deep learning approach to predict focal visual field progression

Integrating a Multi-label Deep Learning Approach with Protein Information to Compare Bioactive Peptides in Brain and Plasma.

Peptide therapeutics is gaining momentum. Advances in the field of peptidomics have enabled researchers to harvest vital information from various organisms and tissue types concerning peptide existence, expression and function. The development of mass spectrometry techniques for high-throughput peptide quantitation has paved the way for the identification and discovery of numerous known and novel peptides. Though much has been achieved, scientists are still facing difficulties when it comes to reducing the search space of the large mass spectrometry-generated peptidomics datasets and focusing on the subset of functionally relevant peptides. Moreover, there is currently no straightforward way to analytically compare the distributions of bioactive peptides in distinct biological samples, which may reveal much useful information when seeking to characterize tissue- or fluid-specific peptidomes. In this chapter, we demonstrate how to identify, rank, and compare predicted bioactive peptides and bioactivity distributions from extensive peptidomics datasets. To aid this task, we utilizeMultiPep, a multi-label deep learning approach designed for classifying peptide bioactivities, to identify bioactive peptides. The predicted bioactivities are synergistically combined with protein information from the UniProt database, which assist in navigating through the jungle of putative therapeutic peptides and relevant peptide leads.

Missing multi-label learning based on the fusion of two-level nonlinear mappings

The relationship between features and labels is an important factor affecting multi-label learning. Many existing multi-label learning approaches decompose label space into a low-rank latent label space and construct linear mappings between the feature space and the latent label space. This type of models may not depict the true data distribution well in high-dimensional spaces. In this work, a model called missing multi-label learning based on the fusion of two-level nonlinear mappings (MML-TLNM) is proposed. Specifically, in the second-level mapping, the label matrix is nonlinearly decomposed into a low-rank latent label matrix and a correlation coefficient matrix to realize the nonlinear transformation from the latent label space to the actual label space. In the first-level mapping, a nonlinear relationship is constructed from the sample matrix to the latent label matrix to realize the nonlinear transformation between the original feature space and the latent label space. The two levels of nonlinear mappings are fused with each other in a nested manner and better depict the true distribution of multi-label data. In addition, the F-norm regularization constraints are imposed on the variables in the two-level nonlinear mappings. Extensive comparative experiments showed that the proposed learning model has good robustness and is superior to the state-of-the-art multi-label learning methods.

Determination of Multi-Component Failure in Automotive System Using Deep Learning

Abstract The connectivity of modern vehicles allows for the monitoring and analysis of a large amount of sensor data from vehicles during their normal operations. In recent years, there has been a growing interest in utilizing this data for the purposes of predictive maintenance. In this paper, a multi-label transfer learning approach is proposed using 14 different pretrained convolutional neural networks retrained with engine simulation data to predict the failure conditions of a selected set of engine components. The retrained classifier networks are designed such that concurrent failure modes of an exhaust gas recirculation, compressor, intercooler, and fuel injectors of a four-cylinder diesel engine can be identified. Time-series simulation data of various failure conditions, which include performance degradation, are generated to retrain the classifier networks to predict which components are failing at any given time. The test results of the retrained classifier networks show that the overall classification performance is good, with the normalized value of mean average precision varying from 0.6 to 0.65 for most of the retrained networks. To the best of the authors’ knowledge, this work represents the first attempt to characterize such time-series data utilizing a multi-label deep learning approach.

Self-paced multi-label co-training

Multi-label learning aims to solve classification problems where instances are associated with a set of labels. In reality, it is generally easy to acquire unlabeled data but expensive or time-consuming to label them, and this situation becomes more serious in multi-label learning as an instance needs to be annotated with several labels. Hence, semi-supervised multi-label learning approaches emerge as they are able to exploit unlabeled data to help train predictive models. This work proposes a novel approach called Self-paced Multi-label Co-Training (SMCT). It leverages the well-known co-training paradigm to iteratively train two classifiers on two views of a dataset and communicate one classifier’s predictions on unlabeled data to augment the other’s training set. As pseudo labels may be false in iterative training, self-paced learning is integrated into SMCT to rectify false pseudo labels and avoid error accumulation. Concretely, the multi-label co-training model in SMCT is formulated as an optimization problem by introducing latent weight variables of unlabeled instances. It is then solved via an alternative convex optimization algorithm. Experimental evaluations are carried out based on six benchmark multi-label datasets and three metrics. The results demonstrate that SMCT is very competitive in each setting when compared with five state-of-the-art methods.

Transfer reinforcement learning method with multi-label learning for compound fault recognition

In complex working site, bearings used as the important part of machine, could simultaneously have faults on several positions. Consequently, multi-label learning approach considering fully the correlation between different faulted positions of bearings becomes the popular learning pattern. Deep reinforcement learning (DRL) combining the perception ability of deep learning and the decision-making ability of reinforcement learning, could be adapted to the compound fault diagnosis while having a strong ability extracting the fault feature from the raw data. However, DRL is difficult to converge and easily falls into the unstable training problem. Therefore, this paper integrates the feature extraction ability of DRL and the knowledge transfer ability of transfer learning (TL), and proposes the multi-label transfer reinforcement learning (ML-TRL). In detail, the proposed method utilizes the improved trust region policy optimization (TRPO) as the basic DRL framework and pre-trains the fixed convolutional networks of ML-TRL using the multi-label convolutional neural network method. In compound fault experiment, the final results demonstrate powerfully that the proposed method could have the higher accuracy than other multi-label learning methods. Hence, the proposed method is a remarkable alternative when recognizing the compound fault of bearings.

Multi-Label Annotation and Classification of Arabic Texts Based on Extracted Seed Keyphrases and Bi-Gram Alphabet Feed Forward Neural Networks Model

In natural language processing, text classification is a fundamental problem. Multi-label classification of textual data is a challenging topic in text classification where an instance can be associated with more than one label. This paper presents a multi-label annotation and classification methodology for Arabic text data that is not currently classified as multi-label, aiming to analyze and compare the performance of various multi-label learning approaches. The current work includes two phases: The first involves automatic annotation of hotel reviews with more than one label based on the aspects found in the reviews. In this phase, review data instances were automatically annotated as multi-label based on the extracted seed keyphrases clusters. The second phase involves experiments to compare the performance of various multi-label classification learning methods. In this phase, we introduced different models including a feed-forward networks model that learns a vector representation based on the bi-gram alphabet rather than the commonly used bag-of-words model. The bi-gram alphabet vector representation model has the advantage of having reduced feature dimensions and not requiring natural language processing tools. The results indicated that employing the bi-gram alphabet vector representation feed forward neural network is a competitive solution for the multi-label text classification problem. It has achieved an accuracy of about 75.2%, and standard deviation (0.062).

A label distribution manifold learning algorithm

In this paper, we propose a novel label distribution manifold learning (LDML) method for solving the multilabel distribution learning problem. First, using manifold learning, we extract the accurate and reduced-dimension features of the training data. Second, we estimate the unknown label distributions associated with the extracted reduced-dimension features based on multi-output kernel regression. Third, we use the extracted reduced-dimension features and their associated estimated label distributions to form an enhanced maximum entropy model, which enables us to accurately and efficiently estimate the unknown true label distributions for the training data. We refer to this algorithm as the LDML. We also propose to apply the tangent space alignment regression in the second stage, and the resulting algorithm is called the LDML-R. The LDML-R has better label distribution learning performance than the LDML but imposes higher complexity than the latter. We evaluate the proposed LDML and LDML-R algorithms on 15 real-world data sets with ground-truth label distributions, and the experimental results obtained show that our method has advantages in terms of learning accuracy compared to the latest multi-label distribution learning approaches. We also use another 10 real-world multi-class data sets, which do not have the ground-truth label distributions, to demonstrate the superior multilabel classification performance of our LDML-R algorithm over the existing state-of-the-art multi-label classification algorithms.

Can Multi-Label Classifiers Help Identify Subjectivity? A Deep Learning Approach to Classifying Cognitive Presence in MOOCs

This paper investigates using multi-label deep learning approach to extending the understanding of cognitive presence in MOOC discussions. Previous studies demonstrate the challenges of subjectivity in manual categorisation methods. Training automatic single-label classifiers may preserve this subjectivity. Using a triangulation approach, we developed a multi-label, fine-tuning BERT classifier to analyse cognitive presence to enrich results with state-of-the-art, single-label classifiers. We trained the multi-label classifiers on the MOOC discussion messages that were categorised into the same phase of cognitive presence by the expert coders, and tested the best-performing classifiers on the messages that the coders categorised into different phases. The results suggest that multi-label classifiers slightly outperformed the single-label classifiers, and the multi-label classifiers predicted the discussion messages as either one category or two adjacent categories of cognitive presence. No messages were tagged as non-adjacent categories by the multi-label classifier. This is an improvement compared to manual categorisation by our expert coders, who obtained non-adjacent categories and even three categories of cognitive presence in one message. In addition to the fully correct prediction, parts of messages were partially correctly predicted by the multi-label classifier. We report an in-depth quantitative and qualitative analysis of these messages in the paper. The automatic categorisation results suggest that the multi-label classifiers have the potential to help educators and researchers identify research subjectivity and tolerate the multiplicity in cognitive presence categorisation. This study contributes to extending the literature on understanding cognitive presence in MOOC discussions.

Improving Large-Scale k-Nearest Neighbor Text Categorization with Label Autoencoders

In this paper, we introduce a multi-label lazy learning approach to deal with automatic semantic indexing in large document collections in the presence of complex and structured label vocabularies with high inter-label correlation. The proposed method is an evolution of the traditional k-Nearest Neighbors algorithm which uses a large autoencoder trained to map the large label space to a reduced size latent space and to regenerate the predicted labels from this latent space. We have evaluated our proposal in a large portion of the MEDLINE biomedical document collection which uses the Medical Subject Headings (MeSH) thesaurus as a controlled vocabulary. In our experiments we propose and evaluate several document representation approaches and different label autoencoder configurations.

Stable Label-Specific Features Generation for Multi-Label Learning via Mixture-Based Clustering Ensemble

Multi-label learning deals with objects associated with multiple class labels, and aims to induce a predictive model which can assign a set of relevant class labels for an unseen instance. Since each class might possess its own characteristics, the strategy of extracting label-specific features has been widely employed to improve the discrimination process in multi-label learning, where the predictive model is induced based on tailored features specific to each class label instead of the identical instance representations. As a representative approach, LIFT generates label-specific features by conducting clustering analysis. However, its performance may be degraded due to the inherent instability of the single clustering algorithm. To improve this, a novel multi-label learning approach named SENCE (stable label-Specific features gENeration for multi-label learning via mixture-based Clustering Ensemble) is proposed, which stabilizes the generation process of label-specific features via clustering ensemble techniques. Specifically, more stable clustering results are obtained by firstly augmenting the original instance representation with cluster assignments from base clusters and then fitting a mixture model via the expectation-maximization (EM) algorithm. Extensive experiments on eighteen benchmark data sets show that SENCE performs better than LIFT and other well-established multi-label learning algorithms.

Low rank label subspace transformation for multi-label learning with missing labels

Multi-label datasets often contain label information with missing values and recovering them is a non-trivial challenge. Several methods augment the observed label matrix by constructing auxiliary labels and learning high order label correlations. Some other techniques exploit the low rank of the label matrix to capture a mix of label correlations. Both these approaches rely on label correlations, however in different ways. In this paper, we propose a unified framework that captures the label correlations utilizing both auxiliary label matrix and the low rank constraints on estimated labels. Our model also enforces maximal separation among different label subspaces for better label differentiation. The proposed method captures local and global correlations using Low Rank label subspace transformation for Multi-label learning with Missing Labels (LRMML). The model considers an auxiliary label matrix which facilitates the missing label information recovery. Low rank on predictions ensures that local label structures are captured and the maximal inter-label subspace separation helps identify discriminatory label correlations. The proposed method builds a multi-label classification model by solving a multivariate difference of convex objective function using surrogate optimization technique and alternating minimization. Empirical results on several benchmark datasets validate the effectiveness of the proposed method against state-of-the-art multi-label learning approaches.

Abstract PD11-02: Subtyping invasive carcinomas and high-risk lesions for machine learning based breast pathology

Abstract Background. The female mammary gland can develop a myriad of epithelial proliferative lesions including, high risk lesions, in-situ and invasive carcinomas. Identification of these pre-neoplastic and neoplastic conditions in biopsy specimens is crucial for proper patient management and may sometimes pose diagnostic challenges for pathologists. Recent research has shown that machine learning algorithms applied to whole slide images (WSI) can accurately detect and grade various cancers; herein, we devise and test a system that classifies the most common preneoplastic and neoplastic conditions of the female breast from WSIs. Design. De-identified slides were scanned on Leica AT2 whole slide scanners (20x; 0.5 µm/pixel) from MSK database were retrieved. Clinical diagnostic metadata were extracted from the pathology reports. Using a multi-label multiple-instance learning (ML-MIL) approach, a SE-ResNet50 Convolutional Neural Network (CNN) was trained to classify atypical lobular hyperplasia (ALH), atypical ductal hyperplasia (ADH), lobular carcinoma in situ (LCIS), ductal carcinoma in situ (DCIS), invasive lobular carcinoma (ILC), invasive ductal carcinoma (IDC). In additional morphological subtypes including apocrine, mucinous, solid papillary, micropapillary, and tubular carcinoma were trained. The system uses the WSI as an input and outputs a slide level class and heatmap for the presence of the trained classes. A validation dataset separate from the training set was used to assess performance of the trained model. Results. The CNN was trained on 9,751 surgical specimens (biopsy, 6,289; excision, 3,462) comprising 40,637 slides. The system was validated on 3,183 breast specimens (biopsy, 1,934; excision, 1,249) comprising 11,447 digital slides that were not included in the training of the CNN model. Validation performance in terms of Area Under Receiver Operating Characteristic Curve (AUROC) for each class is shown in Table 1. Conclusion. The trained CNN had a high performance in identifying the presence of ADH, ALH, DCIS, IDC, ILC, LCIS, and, apocrine, micropapillary, mucinous, solid papillary, and tubular carcinomas. Further studies expanding classes to include all clinically relevant lesions and morphologies are underway. In addition, the same approach can be used to detect microinvasions and calcifications in breast tissue. Table 1.Area Under Receiver Operating Characteristic Curve for Breast Lesion ClassesClassNum. Positive (specimens)Num. Negative (specimens)AUROCADH24718640.903ALH20918860.950LCIS17520080.958DCIS81920920.956IDC52126620.956ILC7131120.934Apocrine2431590.931Micropapillary17230110.927Mucinous1231710.994Solid Papillary1531680.908Tubular carcinoma831750.990 Citation Format: Matthew G Hanna, Patricia Raciti, Alican Bozkurt, Ran Godrich, Julian Viret, Donghun Lee, Philippe Mathieu, Matthew Lee, Eugene Vorontsov, Tomer Sabo, Felipe C Geyer, Jorge S Reis-Filho, Leo Grady, Thomas Fuchs, Christopher Kanan. Subtyping invasive carcinomas and high-risk lesions for machine learning based breast pathology [abstract]. In: Proceedings of the 2021 San Antonio Breast Cancer Symposium; 2021 Dec 7-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2022;82(4 Suppl):Abstract nr PD11-02.

Improving Multi-Label Learning by Correlation Embedding

In multi-label learning, each object is represented by a single instance and is associated with more than one class labels, where the labels might be correlated with each other. As we all know, exploiting label correlations can definitely improve the performance of a multi-label classification model. Existing methods mainly model label correlations in an indirect way, i.e., adding extra constraints on the coefficients or outputs of a model based on a pre-learned label correlation graph. Meanwhile, the high dimension of the feature space also poses great challenges to multi-label learning, such as high time and memory costs. To solve the above mentioned issues, in this paper, we propose a new approach for Multi-Label Learning by Correlation Embedding, namely MLLCE, where the feature space dimension reduction and the multi-label classification are integrated into a unified framework. Specifically, we project the original high-dimensional feature space to a low-dimensional latent space by a mapping matrix. To model label correlation, we learn an embedding matrix from the pre-defined label correlation graph by graph embedding. Then, we construct a multi-label classifier from the low-dimensional latent feature space to the label space, where the embedding matrix is utilized as the model coefficients. Finally, we extend the proposed method MLLCE to the nonlinear version, i.e., NL-MLLCE. The comparison experiment with the state-of-the-art approaches shows that the proposed method MLLCE has a competitive performance in multi-label learning.

Multi-Keyword Classification: A Case Study in Finnish Social Sciences Data Archive

In this paper, we consider the task of assigning relevant labels to studies in the social science domain. Manual labelling is an expensive process and prone to human error. Various multi-label text classification machine learning approaches have been proposed to resolve this problem. We introduce a dataset obtained from the Finnish Social Science Archive and comprised of 2968 research studies’ metadata. The metadata of each study includes attributes, such as the “abstract” and the “set of labels”. We used the Bag of Words (BoW), TF-IDF term weighting and pretrained word embeddings obtained from FastText and BERT models to generate the text representations for each study’s abstract field. Our selection of multi-label classification methods includes a Naive approach, Multi-label k Nearest Neighbours (ML-kNN), Multi-Label Random Forest (ML-RF), X-BERT and Parabel. The methods were combined with the text representation techniques and their performance was evaluated on our dataset. We measured the classification accuracy of the combinations using Precision, Recall and F1 metrics. In addition, we used the Normalized Discounted Cumulative Gain to measure the label ranking performance of the selected methods combined with the text representation techniques. The results showed that the ML-RF model achieved a higher classification accuracy with the TF-IDF features and, based on the ranking score, the Parabel model outperformed the other methods.

Identifying multi-functional bioactive peptide functions using multi-label deep learning.

The bioactive peptide has wide functions, such as lowering blood glucose levels and reducing inflammation. Meanwhile, computational methods such as machine learning are becoming more and more important for peptide functions prediction. Most of the previous studies concentrate on the single-functional bioactive peptides prediction. However, the number of multi-functional peptides is on the increase; therefore, novel computational methods are needed. In this study, we develop a method MLBP (Multi-Label deep learning approach for determining the multi-functionalities of Bioactive Peptides), which can predict multiple functions including anti-cancer, anti-diabetic, anti-hypertensive, anti-inflammatory and anti-microbial simultaneously. MLBP model takes the peptide sequence vector as input to replace the biological and physiochemical features used in other peptides predictors. Using the embedding layer, the dense continuous feature vector is learnt from the sequence vector. Then, we extract convolution features from the feature vector through the convolutional neural network layer and combine with the bidirectional gated recurrent unit layer to improve the prediction performance. The 5-fold cross-validation experiments are conducted on the training dataset, and the results show that Accuracy and Absolute true are 0.695 and 0.685, respectively. On the test dataset, Accuracy and Absolute true of MLBP are 0.709 and 0.697, with 5.0 and 4.7% higher than those of the suboptimum method, respectively. The results indicate MLBP has superior prediction performance on the multi-functional peptides identification. MLBP is available at https://github.com/xialab-ahu/MLBP and http://bioinfo.ahu.edu.cn/MLBP/.

Generic Multi-label Annotation via Adaptive Graph and Marginalized Augmentation

Multi-label learning recovers multiple labels from a single instance. It is a more challenging task compared with single-label manner. Most multi-label learning approaches need large-scale well-labeled samples to achieve high accurate performance. However, it is expensive to build such a dataset. In this work, we propose a generic multi-label learning framework based on Adaptive Graph and Marginalized Augmentation (AGMA) in a semi-supervised scenario. Generally speaking, AGMA makes use of a small amount of labeled data associated with a lot of unlabeled data to boost the learning performance. First, an adaptive similarity graph is learned to effectively capture the intrinsic structure within the data. Second, marginalized augmentation strategy is explored to enhance the model generalization and robustness. Third, a feature-label autoencoder is further deployed to improve inferring efficiency. All the modules are jointly trained to benefit each other. State-of-the-art benchmarks in both traditional and zero-shot multi-label learning scenarios are evaluated. Experiments and ablation studies illustrate the accuracy and efficiency of our AGMA method.

A Multi-instance Multi-label Dual Learning Approach for Video Captioning

Video captioning is a challenging task in the field of multimedia processing, which aims to generate informative natural language descriptions/captions to describe video contents. Previous video captioning approaches mainly focused on capturing visual information in videos using an encoder-decoder structure to generate video captions. Recently, a new encoder-decoder-reconstructor structure was proposed for video captioning, which captured the information in both videos and captions. Based on this, this article proposes a novel multi-instance multi-label dual learning approach (MIMLDL) to generate video captions based on the encoder-decoder-reconstructor structure. Specifically, MIMLDL contains two modules: caption generation and video reconstruction modules. The caption generation module utilizes a lexical fully convolutional neural network (Lexical FCN) with a weakly supervised multi-instance multi-label learning mechanism to learn a translatable mapping between video regions and lexical labels to generate video captions. Then the video reconstruction module synthesizes visual sequences to reproduce raw videos using the outputs of the caption generation module. A dual learning mechanism fine-tunes the two modules according to the gap between the raw and the reproduced videos. Thus, our approach can minimize the semantic gap between raw videos and the generated captions by minimizing the differences between the reproduced and the raw visual sequences. Experimental results on a benchmark dataset demonstrate that MIMLDL can improve the accuracy of video captioning.

Label Enhancement for Label Distribution Learning

Label distribution is more general than both single-label annotation and multi-label annotation. It covers a certain number of labels, representing the degree to which each label describes the instance. The learning process on the instances labeled by label distributions is called label distribution learning (LDL). Unfortunately, many training sets only contain simple logical labels rather than label distributions due to the difficulty of obtaining the label distributions directly. To solve this problem, one way is to recover the label distributions from the logical labels in the training set via leveraging the topological information of the feature space and the correlation among the labels. Such process of recovering label distributions from logical labels is defined as label enhancement (LE), which reinforces the supervision information in the training sets. This paper proposes a novel LE algorithm called Graph Laplacian Label Enhancement (GLLE). Experimental results on one artificial dataset and fourteen real-world LDL datasets show clear advantages of GLLE over several existing LE algorithms. Furthermore, experimental results on eleven multi-label learning datasets validate the advantage of GLLE over the state-of-the-art multi-label learning approaches.

Global-Local Label Correlation for Partial Multi-Label Learning

Partial Multi-label Learning (PML) addresses the scenario where each instance is assigned with multiple candidate labels, while only a subset of the labels are relevant. This task is very challenging because the training procedure can be misguided by the noisy (irrelevant) labels. Exploiting label correlations is useful for partial multi-label learning. However, the existing PML methods often ignore to explicitly and sufficiently leverage the label correlation information for handling the noisy labels. To this end, in this paper, we propose a novel <i>Global-Local Label Correlation (GLC) approach for partial multi-label learning</i>. On one hand, we introduce a label coefficient matrix to explicitly exploit the global structure information of labels from multiple subspaces. On the other hand, we present a new label manifold regularizer to capture the local label correlations to further improve the performance of our method. By jointly taking advantage of the global and local label correlations, our proposed approach achieves superior performance on both the synthetic and real-world data sets from diverse domains.